Introduction

This document records the construction of the Severity Index.

To be able to reach, properly assess and understand local dynamics, vulnerabilities and capacities of the displaced and host populations alike, humanitarian organisations are increasingly using sub-national Area Based Approach. Area based approaches define “an area, rather than a sector or target group, as a primary entry point”. Such approach is particularly appropriate when residents in an affected area face complex, inter-related and multisectoral needs, resulting in risk of forced displacement. Severity index informs therfore comparison of needs across geographic areas. The challenge is to summarize and condense the information of a plurality of underlying indicators into a single measure, in a way that accurately reflects the underlying concept.

The approach used here generates Transparency, Robustness & Accuracy as it follows the standards 10 steps in a fully reproducible approach, using coinR package. It also ensures Credibility & Engagement through the effective participation of field experts as the last stage of the process.

Conceptual Framework

The “theoretical” conceptual framework of the Severity Index is defined as shown in the figure below [NOTE: to ensure this is the final version?]:

Theoretical conceptual framework

However, based on the indicators available in the country and that have been used as the input here, the conceptual framework used in practice is shown in the interactive diagram below.

You can click on the diagram to drill down the different elements of the index.

This may differ from the theoretical version in cases where no indicators are available for certain categories or dimensions, in which case these are automatically removed.

Some further details summarising the input data are given below. These should be checked to ensure they agree with expectations.

#> ----------
#> Your data:
#> ----------
#> Input:
#>   Units: 340 (GT0101, GT0102, GT0103, ...)
#>   Indicators: 41 (InsegAlim, Dormitoria, VivCollectCalle, ...)
#> 
#> Structure:
#>   Level 1 Indicator: 41 indicators (DesastresNat, Deportadas, Refugiados, ...) 
#>   Level 2 Category: 12 groups (Desastres, Desplaz, Violencia, ...) 
#>   Level 3 Dimension: 3 groups (Amenazas, Cap_Resp, Sit_SocEc) 
#>   Level 4 Index: 1 groups (MVI)

Indicator analysis

In this section indicators are analysed from a statistical perspective. Specifically the following analyses are run:

  1. Univariate analysis based on data availability, uniqueness, and distribution properties.
  2. Bivariate analysis based on correlations.

This is automated in a single function which flags any indicators that have “problems”, based on the following criteria:

  • Percentage missing data (indicators flagged with less than 66%)
  • Percentage unique values (any indicators with greater than 50% observations with the same value are flagged)
  • Presence of outliers, flagged by having both absolute skew > 2 and kurtosis > 3.5
  • Collinearity with other indicators in the same aggregation group (defined as a rank correlation > 0.9)
  • Strong negative correlation with other indicators in the same group (defined as a rank correlation < -0.4)

Running the analysis generates the following table:

In the table, only indicators with at least one “flag” based on the criteria mentioned are listed. In order to dig further into the flagged indicators, we can plot individual distributions: this can show the nature of outliers that are flagged.

To explore other indicators change iCode in the source code to the required indicator code. Note that outliers may point to an error, but are often simply a feature of the data. However it is important to check. In the next section, any distributions with outliers will be treated.

Similarly we can explore the relationships between indicators using scatter plots, and this helps to check any collinear or negatively-correlated indicators as flagged in the table.

Again, this plot can be changed by changing the indicator codes.

If any indicators are highly problematic (e.g. multiple flags and/or very low data availability) they can be considered for removal using the f_remove_indicators() function.

Index construction

In order to build the index from this point, we follow three steps:

  1. Outlier treatment
  2. Normalisation
  3. Aggregation

In practice this is performed by a single dedicated function f_build_index(). Here, the steps followed inside this function are broken down.

Outlier treatment aims to adjust the distributions of highly skewed, or fat tailed indicators, including cases where there are outliers that are not characteristic of the rest of the distribution. This is done to improve the discriminatory power of the indicator in aggregation. For more on this, see here.

To deal with this we follow a standard procedure which looks like this, for each indicator:

  1. Check the skew and kurtosis of the indicator
  2. If the absolute skew is greater than 2 AND the kurtosis is greater than 3.5, data treatment is applied (step 3 onwards), else leave the indicator as it is and move back to 1 for the next indicator.
  3. Apply Winsorisation up to a maximum number of five points. Check after each Winsorised point whether skew and kurtosis fall back within the limits specified. If so, apply no further data treatment and move to the next indicator.
  4. If the maximum number of Winsorised points is reached and skew and kurtosis are still outside the thresholds, “undo” any Winsorised points and apply a log transformation.

Running f_build_index() performs this outlier treatment procedure automatically to all indicators (any without outliers are skipped).

The details of how each indicator was treated are shown in the following table.

The next step in the construction process is to normalise the indicators. This is done by transforming each indicator onto a \([1,100]\) scale using a “min-max” transformation. The objective is to put indicators on the same scale for the purposes of balanced aggregation. This is a linear transformation as illustrated by a before/after plot of any given indicator:

The final operation is to aggregate the indicators. In this document we consider several alternative aggregation methods representing various “scenarios” for further comparison by field experts. Therefore, details of each aggregation method are left until the following section. Note that the choice of which aggregation methods to include here is still up for discussion.

Aggregation Options

Our first view of the results is as a results table. The table is sorted by default from the highest scoring (most vulnerable) municipalities downwards, based on the Index scores for different scenario.

Scenario 1: Arithmetic mean

These results should be checked to see whether they agree with common sense. Another way of looking at the results is in a bar chart. The colours within each bar represent the contributions to the index score from each Dimension. This bar chart shows the top 30 most vulnerable municipalities.

The final view of the results is as a choropleth map.

Scenario 2: Geometric mean

One issue to address when aggregating indicators is related to the concept of compensability. Namely the question is to know to what extent can we accept that the high score of an indicator should compensate the low score of another indicator? While the arithmetic mean has “perfect compensability”, the geometric mean only allows partial compensation between indicator scores.

Again the top 30 municipalities in this scenario:

The map:

Scenario 3: Benefit of the Doubt

This method is the application of Data Envelopment Analysis (DEA) to the field of composite indicators. It was originally proposed by Melyn and Moesen (1991) to evaluate macroeconomic performance. ACAPS has prepared an excellent note on The use of data envelopment analysis to calculate priority scores in needs assessments.

BoD approach offers several advantages:

  • Weights are endogenously determined by the observed performances and benchmark is not based on theoretical bounds, but it’s a linear combination of the observed best performances.

  • Principle is easy to communicate: since we are not sure about the right weights, we look for “benefit of the doubt” weights (such that your overall relative performance index is as high as possible).

Again the top 30 municipalities in this scenario:

The map:

Scenario 4: Dendric method

Dendric method (also known as the Wroclaw Taxonomic Method ), originally developed at the University of Wroclaw, is based on the distance from a theoretical unit characterized by the best performance for all indicators considered.

The final composite indicator is therefore based on the sum of euclidean distances from the ideal unit and normalized by a measure of variability of these distance (mean + 2*std).

Again the top 30 municipalities in this scenario:

The map:

Comparison

We can compare the aggregation options as follows:

We can also plot any of the ranks of one result against another. For example: the arithmetic mean against the geometric mean.

This demonstrates in this case the fairly large difference between the two methodologies.

Uncertainty Analysis

Uncertainty analysis involves estimating the uncertainty in the the scores and ranks of the composite indicator, given the uncertainties in the methodological alternatives for the aggregation approach.

The results of an uncertainty include for example confidence intervals over the ranks and median ranks.

Here we will perform an uncertainty analysis which tests the effect of changing the aggregation method, and the weights. To do this we read in some weight sets from a spreadsheet that represent plausible alternatives. These are added to the “coin” and can be accessed during aggregation.

We can now build the specifications for the uncertainty analysis.

Now we run the uncertainty analysis. Since this involves regenerating the results many times, it can take some time. Set N to a lower value to make this faster (but less accurate).

Now we can plot the results. First, the uncertainty:

Here the vertical lines represent 90% confidence intervals for the rank of each unit based on the range of ranks observed due to the different combinations of weights and aggregation methods. It demonstrates that the range of uncertainty is quite wide. To investigate further, we can check the relative importance of the two assumptions via a sensitivity analysis:

This shows that the aggregation method is the most important assumption here. However it is also true that the weight sets are rather different from one another.

The outcome is that the level of uncertainty is fairy high. However, this is quite typical when changing between arithmetic and geometric mean. It points to the need to ideally decide whether the concept of “severity” is better dealt with by the arithmetic mean or the geometric mean, because the choice is important. Notably, this analysis excludes the other two aggregation methods which would further inflate the uncertainty.

Field experts “Reality Check”

Composite indicators try to capture complex multidimensional socioeconomic concepts in a single measure. Inevitably this involves making a number of assumptions in the construction process, and this introduces uncertainties.

As with any model, composite indicators should be validated if at all possible. In composite indicators, an important “validation” step is for field experts to carefully check the results to see to what extent they agree with intuition and experience. If the results seem unusual, it is important not to dismiss them but to try to understand why. Is the discrepancy due to the choice of methodology? Or the fact that some important indicators may be missing or not weighted strongly enough? Is the conceptual framework fit for purpose? Or, is this a genuine signal from the data that agrees with the concept and may be revealing something new to the experts? In short, it is important to explore the data carefully down to the indicator level, during the validation step, as opposed to looking only at the index level.

To summarise, the “Reality check” of the results with the Country Panel Expert can include asking:

  • Do the results options make sense to field experts?
  • If the results don’t make sense, why not?
  • Are there any big gaps in terms of indicators measured?
  • Are there need to reshuffles of indicators/categories?

Generally speaking the objective is to improve the index, and to strike a balance between guiding the index towards the results you expect, without adjusting the methodology to only agree with prior assumptions.